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ABSTRACT 

This functional model for managing large-scale ' 
program evaluations was developed and validated in connection with 
the assessment of Tennessee's Nutrition Education and Training 
Program. Management of such a large-scale assessment requires the 
development of a structure for the organization; distribution and 
recovery of large quantities of materials; the hiring, training, and 
supervision of part-time temporary assistants to administer the 
assessment; and scheduling and supervision in a natural^^s^tt ing. 
Three elements of the model are critical to its implementation. The 
first- is the development of a leadership, framework (to establish and 
clarify lines of responsibility among members of the assessment 
staff). The second is construction of an activity timeline 
(containing personnel assignments and target "dates for completion). 
The third element is a structured procedure for the recruitment, 
interviewing, selection, and training of field assistants (including 
determining job qualifications, formulating screening instruments and 
interview questions, and developing a handbook for field assistants). 
Other important elements of the model include a representative 
sampling procedure, communication channels with cooperating agencies, 
an assessment schedule, and instrumentation and data-gathering 
procedures. Also important are assignment of field assistants to 
specific sites and solicitation of feedback. (Author/JM) ^ 
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A FUNCTIONAL Mcfr)EL FOR MANAGEMENT OF LARGE SCALE ASSESSMENTS 

Trudy W. Banta , Wilma Jozwiak Margaret P. McCabe ' 

The University of Tennessee, Knoxville , 

* • ^ Introduction 

The management of a large-scale evaluation presents a number of problems 
related to assessment procedures, such as (a) the development of a structure 
for the organisation, distribution, and recovery* of iarge quantities of materials; 
(b) the hiring, training, and supervision of part-time temporary assistants t,a 
administer the ^assessment; and»(c) the scheduling and supervision of .the actual 
assessment in the 'natural setting. The Tennessee ^Nutrition Education and 
Training (NET) Program evaluation ^component provides an example of such a large- 
scale evaluation project. On the basis of the experience gained' in tfye first 
year of this evaluation a functional model for managing large-scale assessments 
in education and non-education settings wa^ developed. In this paper the model 
and its validation during the second year of the NET project will be described. 

The Tennessee NET Program is a federally funded project which has as 
focus the improvement of nutrition education for school children in the st*t«u 
The Program design includes evaluation ofr (1) training workshops for educatjors, 
food service personnel, and residential child care workers; (2) nutrition j 
education materials--tt}e quality^of these materials, their usability, and th$ir 
usage in .school settings; and (3) nutrition knowledge, attitudes, perceptions, 
and behaviors of students in Grades K-12, their parents and teachers, school 
administrators and food service personnel. This paper will focus on manage- 
ment procedures" utilized in (3) above — the state-wide assessment of nutrition , 
knowledge, attitude#, perceptions, and behaviors. 

Initially the results of a state-wide needs assessment wer'e used by an 
interdisciplinary committee composed of personnel in the College of Home l 
Economics and the Bureau of EducatiQnal Research and Service at the University 
of Tennessee, Kno.xville to develop a set of nutrition education goals and 
objectives for each grade level,' K-12. A training program based on' these goals 
and objectives was developed for teachers and food service personnel. 

Ia order to assess the effectiveness of this training program the evaluation 
team, which included some of the same personnel who developed the goals and 
objectives, designed a series of assessment instruments. A treatment-control 
group evaluation cL^figu was used in which treatment consisted of the training 
program for teams of K-6 teachers and food service managers from participating 
schools. A sample|£>f 48 schools was selected employing stratification to 
insure the incluSicm of schools from all of tpe state's nine development * 
districts as well as from rural, urban, and suburban areas . * Within the 
stratified sajnple schools were assigned randomly to. treatment or control 
conditions. Becritu^e no treatment had been developed for teachers of Grades 7-12, 
the testing of students in these grades served as an additional control mechanism. 

Planning and implementation of the assessment component of Tennessee f Js 
NET evaluation took place over a two-year period. Pretesting of nutrition* 
knowledge, attitqdes, perceptions, and behaviors was conducted during April. ^ 
.1980. Teachers^and food service managers were given most of a school -year to 
utilize their tj^iffiing before posttesting was conducted in April 1981. Detailed 
analysis of the:i^.rst year's experience provided valuable guidance f of the 
second year and^iffesulted in a functional model for the management of largje scale 
assessments* J£* J 



Management of the NET Assessment required ' selecting, training, and super- 
vising 15 temporary part-time personnel to carry out the state-wide testing. 
'These personnel, called field assistants, were required to learn' a set of 
relatively complex tasks (coding instruments, testing- children at different 
developmental levels, conducting food consumption observations, etc.) and to 
carry them out quickly and accurately in an unfamiliar setting. In addition, 
they were expected to develop, and sustain good relationships with key personnel- 
in the schools. Procedures had to be developed for distribution of test 
instruments and retrieval of large amounts of data in a relatively short period - 
of time.. Elements of the management model developed in connection with the 
NET assessment are displayed in Figure 1. Three elements of the model critical 
to its implementation are: 

.the development of a leadership framework through which effective 
' management can take place 

.construction of an activity timeline 

.recruitment, interviewing, selection, and training of field assistants. 

These three elements will^be discussed in detaiL. 

Leadership Framework 

Or\e purpose of. the leadership framework (see Figure 2) is to establish 
and clarify lines of responsibility among members of the' assessment staff. 
For example, the leadership framework developed for the evaluation component 
of Tennessee's NET program included: (1)* an evaluation director responsible 
for overseeing the total assessment; (2) middle level supervisors with * 11 

responsibility for training and scheduling; (3)' a technical assistant to main- 
tain contact with the field assistants, provide materials for testing, and 
act as a source of quality control for the data as.it is returned; ar^l (4) 
clerical personnel to arrange transportation and lodging for field assistants, 

♦complete paperwork for employment and travel, and maintain communication among 
the members of the project. , * ' 

Another reason for developing a leadership framework is to delineate 
lines of communication between an^ among* key personnel on the assessment 
staff. Establishing definite internal channels of communication facilitated 
the decision-making process affecting Tennessee's NET evaluation. Two different 
communication patterns were utilized. These patterns are depicted in Figures 
2 and 3. Figure 2 portrays the lines of communication utilized in planning* 
(a) the activity timeline and (b) activities associated with th£ recruitment, 
selection, and training of field assistants. During planning sessions concerning 
these two activities, free and open communication was encouraged among 'all 
levels of personnel; clerical personnel had input equal to that of the'manage- 
ment X^am in outlining policies and procedures. 

During implementation of these procedures, however, communicatixDn channels 

, were hierarchical in nature (see Figure 3). Field assistants referred questions 
and problems either to the technical assistant or to clerical personnel, 
depending up$>n the nature of the question or problem. Middle level supervisors 
communicated closely with the technical assistant and clerical personnel to 
monitor initial agency contacts^ factors associated with the efficiency of 
data collection and personnel scheduling for the actual assessment; iij addition, 
they handled matters affecting data processing such as the development of 
instructions for ease of data retrieval and initial ^computer programming. 
Middle level supervisor responsibilities for the first year of Tennessee r s NET 
evaluation project were quantitatively and qualitatively different frdm those * 
outlined for the second year. ' Middle level supervisors for the first year of 
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the* project were responsible for developing instrumentation and data processing 
procedures, whereas in the second year of operation they were able to devote 
more time to quality control of the data using standards developed during the 
first year. During all phases of both project years, the evaluation director 
communicated daily with the middle level supervisors. Further, the director 
handled, sensitive -administrative issues such as convincing reluctant principals 
and curriculum specialist^ of the importance of participating in the- study. 

Activity Timeline 

Another element crucial to the implementation of the model is the 
activity timeline. Development of the timeline is the responsibility of the 
evaluation director and the middle level supervisors. These persons function 
as a management team. Initial assignments for other personnel levels are 
made by the management teanu Detailed discussions among the management team 
and other assessment staff members may result in changes in assignments and 
target dates prior' to implementation. Alternative suggestions made by assess- 
ment staff members are considered carefully by the management team; these 
.suggestions may prove to be more realistic than the initial plans made by tht 
management team. Plans should be flexible enough to allow changes in personnel 
assignments when a task becomes more time-consuming than expected or when a 
staff person proves to be more suited to one job than another.* Flexibility in 
activity timeline construction and in attendant assignment of responsibilities 
allows maximal utilization of personnel. 

A more obvious reason for using an activity timeline is to provide for 
efficient management of the time available for the project. One of the pitfall; 
in many evaluation projects is 5 failure to allow sufficient time for completion 
of essential preliminary tasks. Large-scale assessments usually require 
cooperation -from outside agencies in such tasks as printing instruments and 
related materials, delivering materials, and processing data. A timeline is „ 
especially useful .when it is considered a blueprint for coordinating assessment 
staff activities with agencies involved in rendering services. In fact, it is 
helpful to develop a simplified version of the< activity* timeline fo-send to 
cooperating agencies in order to facilitate communication about the schedule 
of assessment activities. * . 

Recruiting, Selecting, and Training Field Assistants 

Another element crucial to successful implementation of the model is a 
structured procedure f or .recruiting, interviewing, selecting, and training 
field assistants. Before rational recruitment and selection decisions can be 
made, the management team must identify desirable characteristics for a field 
assistant in the situation at band. Obvious characteristics include literacy, 
availability during the period of assessment, and ability to do detailed work 
accurately. A state-wid,e -assessment also ,will require the flexibility to 
travel for a few days at a time. The NET Assessment was conducted in , 
elementary schools; the management team took school dress and behavior cfedes 
into consideration, as well as ability to relate to specific children and 
adults, such as those in inner city or rural areas. 

Screening instruments based on these considerations as well as other 
characteristics found to be important during the first year of the project 
(e.g., self-confidence, assertiveness , and problem-solving skills) werq 
developed by the management team. Specific questions designed to reveal 
presence or* absence of these desired characteristics were asked during group 
And individual interviews. During each interview session, management team 
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members rated each applicant independently using^a standard rating scalp 
developed for that purpose. Scores were summed and average^ applicants 
with the highest ratings were hired to fill field assistant positions. 

In recruiting qualified persons to apply for these positions, it is 
advisable to tap several different s6urces. For example, in connect ioni with 
Tennessee* s JjET assessment component applicant names were solicited frons^aeh 
disparate ^TOrces as (a) elementary school principals 1 substitute teacher 
lists; (b) The University of Tennessee Office of Personnel; and (c) a part- » 
time, temporary employment agency. SeveraV^persons also were hired to fill 
field assistant positions on the recommendation of assessment staff members. 

Those applicants selected to act as field assistants in a large-scale 
assessment must receive some level of training prior to conducting the assess- 
ment. The amount of time set aside for training depends on the complexity of 
the procedures and, realistically, the amount of money allotted for training. 
Because the reliability, and thus the validity, of the assessment depends in 
part on the training procedures, it is important to expend appropriate time 
and effort on this aspect of the assessment. 1 

The one-day training procedure used with the NET Assessment was 
intended to introduce the field assistants to all aspects of th.eir jobs. 
Although th£ management team bore major responsibility for planning this 
one-day training session, other assessment staff/ members also were consulted 
during the planning phase.* All members of the assessment staff participated 
in outlining specific assessment procedures and general guidelines' for 
handling problems which might arise. At the end of the training session, 
field assistants were given an opportunity to ask questions or talk about 
any concerns they might have. In addition, procedures for emergency assis- 
tance were discussed. 

During the training session, a handbook was given to each field assistant. 
This handbook, which contained copies of all instruments and answer sheets, 
information about contacting agencies to be assessed, travel arrangements, 
materials pickup and return, assessment procedures, and ^ost-assessment evalu- 
ation, was designed to provide continuing aid in the field. In addition, 
telephone numbers of management team personnel were included so that crisis 
intervention at any hour was possible. / 

- Other important elements connected with implementing this functional 
model for management of large scale assessments include: * 

.development of a representative sampling procedure 
.development and maintenance of communication- channels with 
. cooperating agencies 
.development of an assessment schedule 
.assignment of field assistants to-specific sites 
.development of instrumentation and data-gathering procedures 
.solicitation of feedback from cooperating agencies and field 
assistants. 

Thes£ element$ will be discussed in the remainder of the paper. 

Sampling Procedures 
• % 

Procedures must be, developed which will ensure a representative 
sample. Since methods of selecting a representative sample are unique to each 
assessment project, sampling procedures will not' be discussed in detail 
except in connection with project management. Sampling procedures for the NET 
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proj*£ct were- developed by an interdisciplinary team of consultants. Selection 
of cooperating schools vira.s a major asp.ect of' the project's first year. Keeping 
this selected sampLe intact during the second .year of assessment depended 
largely upon developing and maintaining effective channels of communication 
with cooperating agencies. 

^ Developing and Maintaining Channels oj: Communication 

In a Large-scale assessment the development and maintenance of communica- 
tion channels between assessment staff members and cooperating agencies is 
essential to implementation of the mode^l. Cooperating agencies must be made 
aware of the crucial part they play in the success of the assessment. In many 
instances, participation of cooperating agencies is voluntary; even when 'it is 
not, courteous and respectful treatment of agency personnel is critical. 

Initial contacts with agencies can be of utmost importance.- In some 
situations, contacting the wrong person in the administrative hierarchy can 
cause* insurmountable problems A general" rule is that an initial contact with 
the person at the top of the hierarchy is best. Subsequent contacts should 
be with the participating individuals. Changes in scheduling must be \ 
communicated to the cooperating agdncy at the earliest possible time. Field 
assistants* should be alerted to the possibility of communication problems 
and encouraged to refer any complaints about the assessment to the management 
team, .who must in turn follow up on v^he^eromplaints . 

Developing An Assessment Schedule 

One of the most- challenging aspects of working with cooperating, agencies 
in a large-scale assessment project .is^the. development of an assessment schedul 
During this phase of < the evaluation close communication between assessment 
staff • members and agency personnel must be maintained*. Assessment dates and 
arrangements (such as assuring the collection' o^ parental permission forms 
and insuring space available for testing) must be communicated clearly and 
consistently to cooperating agencies. Further , ^agencies must be allowed to ' 
provide feedback concerning the desirability of scheduled assessment dates. 

In connection with this , scheduling phase^ of the Tennessee NET Assessment 
a structured communication process between assessment staff members and agency 
personnel was established. The need If or such a procedure during the Second 
year of the project was evident from the first year's experience. This 
structured communication process involved contacts at several points during 
the assessment. Contacts included: 

.a letter informing food service supervisors and curriculum specialists 
of the purpose of the project and outlining the proposed assessment 
activities" . " 

.an initial letter to principals of sample schools enlisting their 
participation in the assessment 

.a second written communication to participating school principals 
requesting that they provide (on a standardized form) information 
concerning school size, possible testing sites within the school, 
the names of'teachers willing to participate in the assessment, 
and convenient dates for test administration 

.a third written communication to participating school"principals 
requesting confirmation of an assessment date. 



v 10 



Participating school personnel were encouraged to write on each form any 
questions they had. Assessment staff ' personnel contacted by telephone 
tnose* schools wHose responses were missing or unclear. Clerical personnel 
*were instructed to answer routine questions; .other questions were referred 
to middle level supervisors ^nd in some cases to the evaluation director. 



* Assigning Field Assistants 

Personnel in each participating school also were contacted by the field 
assistant assigned to that school. Fiel3 assistants introduced themselves, 
confirmed the proposed testing date once again, inquired about motel accommo- 
dations near the assessment site, and asked for specific directions to the 
school. Assignment of field assistants to their respectiveischools* is an 
/^ttft^t5rtant task. Field assistants should he assigned to schools 1 where they , 
^ will £e able to establish rapport and a good working relationship with school 
personnel. For example, some/.persons seem to work best in rural settings, 
whereas others are able to interact more effectively in urban areas.^ ^ 

Developing Instrumentation and Data-£athering Proce'dures 

j 

Development of instrumentation and dat^-gathering procedures is another 1 - \ 
element crucial to successful implementation of the model. Important consid- J 
erations in the development of assessment instruments are ease of administration 
and efficiency of data retrieval. Careful, early planning can circumvent 
many problems in these areas. 

As noted previously, lafcge-scale assessments usually must be Conducted 
with the help of temporary, part-time employees. Training for these individuals 
by necessity cannot be extensive. The more straightf orward^ are the procedures 
reflected in instrument design, the more likely it is that administration will 
be consistent. When multiple forms of an instrument will be used, care should 
be taken to make the methods of administration for each as similar as possible. 
Instructions to the respondents wf^ici^appear on the forms should be simple ; 
and clear so that the field assistant is not put in the position of ^having to 
make interpretations which may affect reliability . ( 

Instrument design also is critical in determining the efficiency with 
which data can be retrieved. Retrieval is facilitated if respondents mark 
their answers on optical scan forms which can be re#d by cbmputer and the 
data entered 'directly into computer storage. Wheru some characteristics of* 
the respondent or the subject matter -Wing assessed prevent the use of optical 
scau forms, instruments can be designed to facilitate data £ntry for computer 1 
analysis by the' arrangement of blanks for responses , - the use of numbered 
-esponses corresponding to the card columns in which they will be entered, and \- 
the labeling of response blanks with card column numbers. 

Duplication of instruments can prove to be a major hurdle in a large-scale 
assessment. Determination of the number of pages in each instrument form 
and the number of each form required must be madeas early as possible so that ' 
definite arrangements can be made for duplication, collation, and stapling. 
Very large duplication jobs often must be scheduled clays or weeks in advance 
with commercial concerns. Tf duplication is to be an ih-house operation, it 
is essential to determine the time required for all portions of the job *o 
avoid disruption off* usual job responsibilities as deadlines near and to make 
arrangements for part-time assistance' if it appears to be -desirable . Even the 
duplication facilities ' of a large university can be badly strained by very 
large orders, as was discovered during the NET Assessment. Wherever the 
duplication is to be accomplished, careful oversight and quality control are 
essential to assure th&t copies are clear and readable "and that instrument 
assembly is correct.'* 




When duplication of instruments had been completed, consideration should 
be given to precoding of instruments- wi th as much information as possible. 
Adult aYfa student, instruments for the NET Assessment were precode*! with 
school, grade, and form codes, Precoding .reduces time demands on the test 
administrators and reduces the likelihood of e,rrdrs in coding. If precoding 
ed, care must be take.n to assure that answer sheets and instruments 
come separated. Optical" scan forms can be identified further by writing 
ade and/or adult group for which they are intended in felt tip pen along 
a margin. » 

Having developed, duplicated, assembled, and precoded the instruments, * 
the evaluator must determine procedures for their "distr ibution and recovery. 
A few hours at this point spent planning and organizing materials and develop- 
ing *a record-keeping system can prevent lost hours and unusable ddta later. 
If possible, one person should be assigned solely to the job of/distributing 
-instruments ahd other materials and equipment. That person should develop 
an estimate' of the^numbers of forms of instruments and other materials that 
will be needed on each day of „the assessment period by each field assistant. 
A checklist, signed by the field assistant after an inspection of the contents 
of the assessment package, will not only help assure that all materials *n4 
equipment will be in place when the field assistant ueeds them, but will 
prevent the disappearance of equipment for which no one seems to.be responsible 
As instruments are jchecked'back in, a simple control procedure can t^e used to 
determine that the correct groups were assessed, answer sheets we'^e completed| 
as instructed, and no raw data were left behind in the setting, the rented , 
car, or the hotel room. 

v. Soliciting Feedback 

Another element essential to implementation of the mociel is solicitation 
of feedback from field assistants and cooperating agencies concerning instru- 
mentation and 'data gathering procedures a$ well as other aspects of the assess- 
ment. Feedback receive^ ,f rom participating agencies and field assistants can 
be of great value not only in a formative.'sense by clearing up problems as 
they emerge, but also in a suitimative sense v by providing the foundation for 
improvement of instruments and procedures' for future assessments. Informal' 
types of feedback, already have been mentioned. Equally important are formal 
feedback' procedures planned as part of the assessment. 

Prior to the beginning of the actual 'assessment process, participating 
agencies should be allowed to give feedback about scheduling and other 
relevant issues.. In the NET Assessment, participating .agencies first 

*were asked to provide a list of dates within a specific time frame that would 
not be convenient for assessment. In a second Communication- agencies were 
provided with a ^specific assessment date and asKed to respond if* that d^te 
would not be convenient. 'After tjie assessment , agencies were asked to * 
complete a brief questionnaire abouC the assessment process. Field assistants 

^also were asked to complete such a questionnaire for each setting in which ^ 
they worked. Review of these feedback contacts at the end of the first year ! s 
assessment formed the basts for changes in procedure for the second year. 

Summary and Cone lusions 

In summary; elements essential to implementation of the model for manage- 
ment of large-scale assessments which has been described include: 

.development of a leadership framework through which effective 
management. can take place / 
construction of an, act ivity^time line 
-.recruitment, interviewing selection, and training of fiel 
^ . assistants. 
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.Other important* elements of the model are the following: 

.development of a representative sampling procedure 
.development and maintenance of communication channels with 
cooperating agencies 

.development of* an assessment schedule 
.assignment of field assistants to specific sites 
.development of a representative sampling procedure 
.'development of instrumentation and data-gathering procedures 
. solicitation of feedback from cooperating agencies and field 
assistants. 

s Certain aspects of the development of this model which served to 
Strengthen it include: 

^.testing and validation of management techniques over a two-year period 
.communication and decision-making strategies designed to facilitate 

the flow- of -information and execution of actions 
.assignment 6f each member *of the assessment staff ta definite 

areas of responsibility with opportunities for decision-makfng 

and information input , 

.development of specific procedures for contacting agency personnel 
. to insure open~cqmmupication between assessment staff members and 
^ agency personnel 

.developm^*ntoT?{>e^lf ic strategies for ^recruitment , interviewing,* 
selection, and training of temporary part-time personnel ' 1 

.utilisation of both formal and informal methods for receiving 
feedback from field assistants and agency personnel. - * 

This model has important implications for evaluation research. It is 
designed to be used as a guide for evaluation *of projects 'of all typdB 1 -- not 
just those dealing with educational programs. Industry uses evaluation 
techniques in assessing training programs for personnel as well as job 
performance. Educators have benefitted from the use of management techniques 
to facilitate large-scale evaluation projects. This model has incorporated 
such management concepts as outlining communication and decision-making 
strategies; establishing a definite plan of action 'and enforcing deadlines 
(e.g., activity timeline); placing emphasis on maximum utilization* of personnel; 
and providing a carefully designed program for on-the-job training. With ever- 
decreasing funds for education and social services comes thfe need for mote 
effective techniques for evaluating existing aijfl proposed programs. The 
development of validated models such as this one can help increase the efficiency 
of evaluation projects involving large-scale Assessments^. 
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